It is common in high-performance computing (HPC) systems and other information processing systems for multiple compute nodes to access a shared file system. For example, HPC systems such as supercomputers typically include large numbers of compute nodes that access a parallel file system, distributed file system or other type of cluster file system. A cluster file system as the term is broadly used herein generally allows multiple client devices to share access to files over a network, and will typically comprise a large number of distinct storage nodes.
Well-known examples of cluster file systems include the Lustre file system and distributed file systems such as Hadoop Distributed File System (HDFS). These and other file systems utilized by HPC systems can readily scale to support tens of thousands of clients, petabytes of storage, and hundreds of gigabytes per second of aggregate input-output (IO) throughput.
HPC systems are often configured to utilize remote direct memory access (RDMA) techniques. Such techniques provide an application memory of a compute node with direct access to a memory of a storage node associated with the shared file system. This avoids the need to copy data between application memory and operating system data buffers in the compute node, thereby providing increased throughput and reduced latency. Nonetheless, as HPC systems continue to scale ever larger in terms of processing resources, conventional RDMA techniques are unable to provide the desired level of IO throughput performance. A need therefore exists for improved techniques for implementing RDMA in HPC systems and other types of information processing systems.